The Shape of Learning Curves: A Review.

Learning curves provide insight into the dependence of a learner's generalization performance on the training set size. This important tool can be used for model selection, to predict the effect of more training data, and to reduce the computational complexity of model training and hyperparameter tuning. This review recounts the origins of the term, provides a formal definition of the learning curve, and briefly covers basics such as its estimation. Our main contribution is a comprehensive overview of the literature regarding the shape of learning curves. We discuss empirical and theoretical evidence that supports well-behaved curves that often have the shape of a power law or an exponential. We consider the learning curves of Gaussian processes, the complex shapes they can display, and the factors influencing them. We draw specific attention to examples of learning curves that are ill-behaved, showing worse learning performance with more training data. To wrap up, we point out various open problems that warrant deeper empirical and theoretical investigation. All in all, our review underscores that learning curves are surprisingly diverse and no universal model can be identified.

Improved Generalization in Semi-Supervised Learning: A Survey of Theoretical Results.

Semi-supervised learning is the learning setting in which we have both labeled and unlabeled data at our disposal. This survey covers theoretical results for this setting and maps out the benefits of unlabeled data in classification and regression tasks. Most methods that use unlabeled data rely on certain assumptions about the data distribution. When those assumptions are not met, including unlabeled data may actually decrease performance. For all practical purposes, it is therefore instructive to have an understanding of the underlying theory and the possible learning behavior that comes with it. This survey gathers results about the possible gains one can achieve when using semi-supervised learning as well as results about the limits of such methods. Specifically, it aims to answer the following questions: what are, in terms of improving supervised methods, the limits of semi-supervised learning? What are the assumptions of different methods? What can we achieve if the assumptions are true? As, indeed, the precise assumptions made are of the essence, this is where the survey's particular attention goes out to.

Predicting patient response with models trained on cell lines and patient-derived xenografts by nonlinear transfer learning.

Preclinical models have been the workhorse of cancer research, producing massive amounts of drug response data. Unfortunately, translating response biomarkers derived from these datasets to human tumors has proven to be particularly challenging. To address this challenge, we developed TRANSACT, a computational framework that builds a consensus space to capture biological processes common to preclinical models and human tumors and exploits this space to construct drug response predictors that robustly transfer from preclinical models to human tumors. TRANSACT performs favorably compared to four competing approaches, including two deep learning approaches, on a set of 23 drug prediction challenges on The Cancer Genome Atlas and 226 metastatic tumors from the Hartwig Medical Foundation. We demonstrate that response predictions deliver a robust performance for a number of therapies of high clinical importance: platinum-based chemotherapies, gemcitabine, and paclitaxel. In contrast to other approaches, we demonstrate the interpretability of the TRANSACT predictors by correctly identifying known biomarkers of targeted therapies, and we propose potential mechanisms that mediate the resistance to two chemotherapeutic agents.

Resolution Learning in Deep Convolutional Networks Using Scale-Space Theory.

Resolution in deep convolutional neural networks (CNNs) is typically bounded by the receptive field size through filter sizes, and subsampling layers or strided convolutions on feature maps. The optimal resolution may vary significantly depending on the dataset. Modern CNNs hard-code their resolution hyper-parameters in the network architecture which makes tuning such hyper-parameters cumbersome. We propose to do away with hard-coded resolution hyper-parameters and aim to learn the appropriate resolution from data. We use scale-space theory to obtain a self-similar parametrization of filters and make use of the N-Jet: a truncated Taylor series to approximate a filter by a learned combination of Gaussian derivative filters. The parameter σ of the Gaussian basis controls both the amount of detail the filter encodes and the spatial extent of the filter. Since σ is a continuous parameter, we can optimize it with respect to the loss. The proposed N-Jet layer achieves comparable performance when used in state-of-the art architectures, while learning the correct resolution in each layer automatically. We evaluate our N-Jet layer on both classification and segmentation, and we show that learning σ is especially beneficial when dealing with inputs at multiple sizes.

A Review of Domain Adaptation without Target Labels.

Domain adaptation has become a prominent problem setting in machine learning and related fields. This review asks the question: How can a classifier learn from a source domain and generalize to a target domain? We present a categorization of approaches, divided into, what we refer to as, sample-based, feature-based, and inference-based methods. Sample-based methods focus on weighting individual observations during training based on their importance to the target domain. Feature-based methods revolve around on mapping, projecting, and representing features such that a source classifier performs well on the target domain and inference-based methods incorporate adaptation into the parameter estimation procedure, for instance through constraints on the optimization procedure. Additionally, we review a number of conditions that allow for formulating bounds on the cross-domain generalization error. Our categorization highlights recurring ideas and raises questions important to further research.

PRECISE: a domain adaptation approach to transfer predictors of drug response from pre-clinical models to tumors.

MOTIVATION: Cell lines and patient-derived xenografts (PDXs) have been used extensively to understand the molecular underpinnings of cancer. While core biological processes are typically conserved, these models also show important differences compared to human tumors, hampering the translation of findings from pre-clinical models to the human setting. In particular, employing drug response predictors generated on data derived from pre-clinical models to predict patient response remains a challenging task. As very large drug response datasets have been collected for pre-clinical models, and patient drug response data are often lacking, there is an urgent need for methods that efficiently transfer drug response predictors from pre-clinical models to the human setting. RESULTS: We show that cell lines and PDXs share common characteristics and processes with human tumors. We quantify this similarity and show that a regression model cannot simply be trained on cell lines or PDXs and then applied on tumors. We developed PRECISE, a novel methodology based on domain adaptation that captures the common information shared amongst pre-clinical models and human tumors in a consensus representation. Employing this representation, we train predictors of drug response on pre-clinical data and apply these predictors to stratify human tumors. We show that the resulting domain-invariant predictors show a small reduction in predictive performance in the pre-clinical domain but, importantly, reliably recover known associations between independent biomarkers and their companion drugs on human tumors. AVAILABILITY AND IMPLEMENTATION: PRECISE and the scripts for running our experiments are available on our GitHub page (https://github.com/NKI-CCB/PRECISE). SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Template Matching via Densities on the Roto-Translation Group.

We propose a template matching method for the detection of 2D image objects that are characterized by orientation patterns. Our method is based on data representations via orientation scores, which are functions on the space of positions and orientations, and which are obtained via a wavelet-type transform. This new representation allows us to detect orientation patterns in an intuitive and direct way, namely via cross-correlations. Additionally, we propose a generalized linear regression framework for the construction of suitable templates using smoothing splines. Here, it is important to recognize a curved geometry on the position-orientation domain, which we identify with the Lie group SE(2): the roto-translation group. Templates are then optimized in a B-spline basis, and smoothness is defined with respect to the curved geometry. We achieve state-of-the-art results on three different applications: detection of the optic nerve head in the retina (99.83 percent success rate on 1,737 images), of the fovea in the retina (99.32 percent success rate on 1,616 images), and of the pupil in regular camera images (95.86 percent on 1,521 images). The high performance is due to inclusion of both intensity and orientation features with effective geometric priors in the template matching. Moreover, our method is fast due to a cross-correlation based matching approach.

Dissimilarity-Based Ensembles for Multiple Instance Learning.

In multiple instance learning, objects are sets (bags) of feature vectors (instances) rather than individual feature vectors. In this paper, we address the problem of how these bags can best be represented. Two standard approaches are to use (dis)similarities between bags and prototype bags, or between bags and prototype instances. The first approach results in a relatively low-dimensional representation, determined by the number of training bags, whereas the second approach results in a relatively high-dimensional representation, determined by the total number of instances in the training set. However, an advantage of the latter representation is that the informativeness of the prototype instances can be inferred. In this paper, a third, intermediate approach is proposed, which links the two approaches and combines their strengths. Our classifier is inspired by a random subspace ensemble, and considers subspaces of the dissimilarity space, defined by subsets of instances, as prototypes. We provide insight into the structure of some popular multiple instance problems and show state-of-the-art performances on these data sets.

Contrastive Pessimistic Likelihood Estimation for Semi-Supervised Classification.

Improvement guarantees for semi-supervised classifiers can currently only be given under restrictive conditions on the data. We propose a general way to perform semi-supervised parameter estimation for likelihood-based classifiers for which, on the full training set, the estimates are never worse than the supervised solution in terms of the log-likelihood. We argue, moreover, that we may expect these solutions to really improve upon the supervised classifier in particular cases. In a worked-out example for LDA, we take it one step further and essentially prove that its semi-supervised version is strictly better than its supervised counterpart. The two new concepts that form the core of our estimation principle are contrast and pessimism. The former refers to the fact that our objective function takes the supervised estimates into account, enabling the semi-supervised solution to explicitly control the potential improvements over this estimate. The latter refers to the fact that our estimates are conservative and therefore resilient to whatever form the true labeling of the unlabeled data takes on. Experiments demonstrate the improvements in terms of both the log-likelihood and the classification error rate on independent test sets.

Semi-supervised nearest mean classification through a constrained log-likelihood.

We cast a semi-supervised nearest mean classifier, previously introduced by the first author, in a more principled log-likelihood formulation that is subject to constraints. This, in turn, leads us to make the important suggestion to not only investigate error rates of semi-supervised learners but also consider the risk they originally aim to optimize. We demonstrate empirically that in terms of classification error, mixed results are obtained when comparing supervised to semi-supervised nearest mean classification, while in terms of log-likelihood on the test set, the semi-supervised method consistently outperforms its supervised counterpart. Comparisons to self-learning, a standard approach in semi-supervised learning, are included to further clarify the way, in which our constrained nearest mean classifier improves over regular, supervised nearest mean classification.

Diagnostic classification of arterial spin labeling and structural MRI in presenile early stage dementia.

Because hypoperfusion of brain tissue precedes atrophy in dementia, the detection of dementia may be advanced by the use of perfusion information. Such information can be obtained noninvasively with arterial spin labeling (ASL), a relatively new MR technique quantifying cerebral blood flow (CBF). Using ASL and structural MRI, we evaluated diagnostic classification in 32 prospectively included presenile early stage dementia patients and 32 healthy controls. Patients were suspected of Alzheimer's disease (AD) or frontotemporal dementia. Classification was based on CBF as perfusion marker, gray matter (GM) volume as atrophy marker, and their combination. These markers were each examined using six feature extraction methods: a voxel-wise method and a region of interest (ROI)-wise approach using five ROI-sets in the GM. These ROI-sets ranged in number from 72 brain regions to a single ROI for the entire supratentorial brain. Classification was performed with a linear support vector machine classifier. For validation of the classification method on the basis of GM features, a reference dataset from the AD Neuroimaging Initiative database was used consisting of AD patients and healthy controls. In our early stage dementia population, the voxelwise feature-extraction approach achieved more accurate results (area under the curve (AUC) range = 86 - 91%) than all other approaches (AUC = 57 - 84%). Used in isolation, CBF quantified with ASL was a good diagnostic marker for dementia. However, our findings indicated only little added diagnostic value when combining ASL with the structural MRI data (AUC = 91%), which did not significantly improve over accuracy of structural MRI atrophy marker by itself.

Femoral cartilage segmentation in knee MRI scans using two stage voxel classification.

Using more than one classification stage and exploiting class population imbalance allows for incorporating powerful classifiers in tasks requiring large scale training data, even if these classifiers scale badly with the number of training samples. This led us to propose a two-stage classifier for segmenting tibial cartilage in knee MRI scans combining nearest neighbor classification and support vector machines (SVMs). Here we apply it to femoral cartilage segmentation. We describe the similarities and differences between segmenting these two knee cartilages. For further speeding up batch SVM training, we propose loosening the stopping condition in the quadratic program solver before considering moving on to other approximation techniques such as online SVMs. The two-stage approach reached a higher accuracy in comparison to the one-stage state-of-the-art method. It also achieved better inter-scan segmentation reproducibility when compared to a radiologist as well as the current state-of-the-art method.

Shape-based assessment of vertebral fracture risk in postmenopausal women using discriminative shape alignment.

RATIONALE AND OBJECTIVES: Risk assessment of future osteoporotic vertebral fractures is currently based mainly on risk factors, such as bone mineral density, age, prior fragility fractures, and smoking. It can be argued that an osteoporotic vertebral fracture is not exclusively an abrupt event but the result of a decaying process. To evaluate fracture risk, a shape-based classifier, identifying possible small prefracture deformities, may be constructed. MATERIALS AND METHODS: During a longitudinal case-control study, a large population of postmenopausal women, fracture free at baseline, were followed. The 22 women who sustained at least one lumbar fracture on follow-up represented the case group. The control group comprised 91 women who maintained skeletal integrity and matched the case group according to the standard osteoporosis risk factors. On radiographs, a radiologist and two technicians independently performed manual annotations of the vertebrae, and fracture prediction using shape features extracted from the baseline annotations was performed. This was implemented using posterior probabilities from a standard linear classifier. RESULTS: The classifier tested on the study population quantified vertebral fracture risk, giving statistically significant results for the radiologist annotations (area under the curve, 0.71 ± 0.013; odds ratio, 4.9; 95% confidence interval, 2.94-8.05). CONCLUSIONS: The shape-based classifier provided meaningful information for the prediction of vertebral fractures. The approach was tested on case and control groups matched for osteoporosis risk factors. Therefore, the method can be considered an additional biomarker, which combined with traditional risk factors can improve population selection (eg, in clinical trials), identifying patients with high fracture risk.

Automated colorectal cancer diagnosis for whole-slice histopathology.

In this study, we propose a computational diagnosis system for detecting the colorectal cancer from histopathological slices. The computational analysis was usually performed on patch level where only a small part of the slice is covered. However, slice-based classification is more realistic for histopathological diagnosis. The developed method combines both textural and structural features from patch images and proposes a two level classification scheme. In the first level, the patches in slices are classified into possible classes (adenomatous, inflamed, cancer and normal) and the distribution of the patches into these classes is considered as the information representing the slices. Then the slices are classified using a logistic linear classifier. In patch level, we obtain the correct classification accuracies of 94.36% and 96.34% for the cancer and normal classes, respectively. However, in slice level, the accuracies of the 79.17% and 92.68% are achieved for cancer and normal classes, respectively.

On combining computer-aided detection systems.

Computer-aided detection (CAD) is increasingly used in clinical practice and for many applications a multitude of CAD systems have been developed. In practice, CAD systems have different strengths and weaknesses and it is therefore interesting to consider their combination. In this paper, we present generic methods to combine multiple CAD systems and investigate what kind of performance increase can be expected. Experimental results are presented using data from the ANODE09 and ROC09 online CAD challenges for the detection of pulmonary nodules in computed tomography scans and red lesions in retinal images, respectively. For both applications, combination results in a large and significant increase in performance when compared to the best individual CAD system.

A texton-based approach for the classification of lung parenchyma in CT images.

In this paper, a texton-based classification system based on raw pixel representation along with a support vector machine with radial basis function kernel is proposed for the classification of emphysema in computed tomography images of the lung. The proposed approach is tested on 168 annotated regions of interest consisting of normal tissue, centrilobular emphysema, and paraseptal emphysema. The results show the superiority of the proposed approach to common techniques in the literature including moments of the histogram of filter responses based on Gaussian derivatives. The performance of the proposed system, with an accuracy of 96.43%, also slightly improves over a recently proposed approach based on local binary patterns.

Image dissimilarity-based quantification of lung disease from CT.

In this paper, we propose to classify medical images using dissimilarities computed between collections of regions of interest. The images are mapped into a dissimilarity space using an image dissimilarity measure, and a standard vector space-based classifier is applied in this space. The classification output of this approach can be used in computer aided-diagnosis problems where the goal is to detect the presence of abnormal regions or to quantify the extent or severity of abnormalities in these regions. The proposed approach is applied to quantify chronic obstructive pulmonary disease in computed tomography (CT) images, achieving an area under the receiver operating characteristic curve of 0.817. This is significantly better compared to combining individual region classifications into an overall image classification, and compared to common computerized quantitative measures in pulmonary CT.

The improbability of harris interest points.

An elementary characterization of the map underlying Harris corners, also known as Harris interest points or key points, is provided. Two principal and basic assumptions made are: 1) Local image structure is captured in an uncommitted way, simply using weighted raw image values around every image location to describe the local image information, and 2) the lower the probability of observing the image structure present in a particular point, the more salient, or interesting, this position is, i.e., saliency is related to how uncommon it is to see a certain image structure, how surprising it is. Through the latter assumption, the axiomatization proposed makes a sound link between image saliency in computer vision on the one hand and, on the other, computational models of preattentive human visual perception, where exactly the same definition of saliency has been proposed. Because of this link, the characterization provides a compelling case in favor of Harris interest points over other approaches.

Breast density changes associated with postmenopausal hormone therapy: post hoc radiologist- and computer-based analyses.

OBJECTIVE: The aim of this study was to assess the impact of oral hormone therapy (HT) on breast density in postmenopausal women and to compare the use of computer-based automated approaches for the assessment of breast density with reference to traditional methods. METHODS: Low-dose oral estrogen (1 mg) continuously combined with drospirenone (2 mg) was administered to postmenopausal women for up to 2 years (26 treatment cycles, 28 d/cycle) in a randomized, placebo-controlled trial. This post hoc analysis assessed the changes in breast density measured from digitized images by two radiologist-based approaches (Breast Imaging Reporting and Data System score and interactive threshold) and one computer-based technique (heterogeneity examination of radiographs). Correlations of temporal changes in breast density with changes in serum estradiol levels, biochemical markers of bone metabolism, and bone mineral density at the spine and femur were also assessed. RESULTS: Breast density assessed by the radiologist-based approaches increased significantly from baseline in the HT group (P < 0.01), with significant divergence from placebo at 2 years (P < 0.01). Heterogeneity examination of radiograph score by computer-based technique was unchanged in the HT group and decreased significantly with placebo (P < 0.001) to produce a significant group divergence (P < 0.05). Changes in mammographic markers by radiologist- and computer-based approaches correlated with each other in the HT group (P < 0.01) but not in the placebo group. CONCLUSIONS: HT for 2 years in postmenopausal women significantly increased radiologist-assessed breast density compared with placebo, in addition to significant changes in estrogen levels, markers of bone metabolism, and bone mineral density. Computer-automated techniques may be comparable with and offer advantages over traditional methods.